SSN _ NLP _ MLRG at S em E val-2020 Task 12: Offensive Language Identification in E nglish, D anish, G reek Using BERT and Machine Learning Approach (2020)

Abstract: Offensive language identification is to detect the hurtful tweets, derogatory comments, swear words on social media. As an emerging growth of social media communication, offensive language detection has received more attention in the last years; we focus to perform the task on English, Danish and Greek. We have investigated which can be effect more on pre-trained models BERT (Bidirectional Encoder Representation from Transformer) and Machine Learning Approaches. Our investigation shows the difference performance between the three languages and to identify the best performance is evaluated by the classification algorithms. In the shared task SemEval-2020, our team SSN_NLP_MLRG submitted for three languages that are Subtasks A, B, C in English, Subtask A in Danish and Subtask A in Greek. Our team SSN_NLP_MLRG obtained the F1 Scores as 0.90, 0.61, 0.52 for the Subtasks A, B, C in English, 0.56 for the Subtask A in Danish and 0.67 for the Subtask A in Greek respectively.

## introduction offensive language detection is the process of identifying and detecting the user generated offensive content or comments that are insult, hurt, profanity, and racism and targeted the individual or group in the massive social media @xcite . as an immense growth of social media, user generated content increasingly occurs on the social media platforms and much more attention to deal with the offensive content. detecting the offensive content is helpful for the field of sentimental analysis @xcite , abusive identification @xcite , aggregation and cyber bullying. several research works have been performed to identify the offensive language in social media. research workshop has reported in nlp communities that are germeval 2018 shared task on the identification of offensive language @xcite . offenseval@semeval2019 @xcite shared task focuses on identification and categorization of offensive content in social media. offenseval@semeval2020 @xcite shared task focuses on multilingual identification and categorization of offensive content for five different languages in social media. recent research area has reported to identifying the offensive content @xcite and abusive comment @xcite in different languages. zeses pitenis (2020) used machine learning and deep learning techniques for the identification of offensive content in greek annotated tweet dataset. @xcite used logistic regression and more different attention in bilstm models to detect and categorize the offensive language in the english and danish annotated facebook, reddit dataset. @xcite used machine learning approaches and sampling method to detect the abusive language in the english and german dataset. our team ssn_nlp_mlrg participated in the shared task offenseval@semeval2020 task 12 for the three languages that are english, greek and danish languages. it focuses on the multilingual offensive language detection, categorization of offensive language and target identification. we have employed traditional machine learning approaches and bert model to identify and categorize the offensive language. the bert model that are implemented for the three subtasks a, b, c of english, subtask a of danish and subtask a of greek. the machine learning approaches are implemented for the subtask b, c of the english language. we have investigated which classifiers show the similarity between the three languages and also find the best performance model for all the subtasks in the three languages. the challenges in the shared task are as follows; a) choosing the limit to set the given text is offensive or not offensive from the average confidence and standard deviation confidence positive values for the three subtasks a, b, c in english language. b) handling of huge data in subtask a in the english language. c) imbalanced data in the subtask a in the danish and greek language. d) removing the noisy data and handling the ungrammatical sentences in all the three languages. e) determine the types of features for the classifier. f) small datasets which is hard to build a complex models for training the data in the subtask a in the danish and greek language. we have used the tf-idf vector for the feature extraction for the classifiers namely linear regression, gaussian naive bayes, support vector machine and random forest. we have investigated the results of the three languages and compared the results of the machine learning approaches with the performance of the pre-trained bert models. the rest of the paper is structured as follows: we discuss background in section 2 and the system overview in section 3. section 4 presents our experimental setup and feature sets. section 5 presents the results, and draws conclusions and discusses future work in section 6. ## system overview we have employed both the traditional machine learning approaches and bert pre-trained models to identify and categorize the offensive language in the english, danish and greek languages. the training dataset for the english subtasks a, b with columns namely, id, text, average, std. id refers the identification number for the tweet, text refers the tweets, and average refers average of the confidences to belong to the positive class for that subtask. the positive class is off for subtask a, and unt for subtask b. std refers confidences standard deviation from avg_conf for a particular instance. the english subtask c with columns namely id, text, avg_ ind, avg_ grp, avg_ oth, std_ind, std_grp, std_oth. avg_ind refers average of the confidences of individual target likewise group target and other target. in the danish and greek languages, the training data with the columns namely id, tweet, subtask_a. here subtask_a refers the labels off and not. first, we have started with the machine learning approaches of the subtask b and subtask c in english language namely logistic regression, multinomial naive bayes (nb), random forest and linear support vector machine (svc) respectively. we have used the doc2vec, tf-idf vectorise for the feature extraction and trained the models using linear regression. doc2vec is not giving good performance in the ungrammatical sentences and tf-idf performs well when compared with doc2vec feature selection. tf-idf vectorise of the english subtask b is (188974, 55585) and the english subtask c is (188973, 55585). we have cross validate the data with the train test split size is 0.25 and maximum number of iterations is 100. second, we have trained the models using multinomial naive bayes, random forest and linear support vector machines. we have cross validated the each model separately. accuracies of the models are shown in the ## experimental setup the test data is given in the tsv file format by the organizer of the semeval 2020 shared task. the format of the test data with the columns namely id, tweet for the three languages. we have preprocessed the data by removing the urls and the text "@user" from the tweets. word tokenizer is used for the subtask a in english, tweet tokenizer is used to obtain the vocabulary and features for the training data. we have used the ntlk libraries and gensim libraries for pre-processing. ## results we have used the evaluation metrics as precision, recall and f1 macro. the performances of the three languages of bert model as shown in the ## conclusion we have implemented both the traditional machine learning and bert pre-trained model to identify the offensive language for the english subtask a, danish subtask a, greek subtask a and categorization of offensive languages for the english subtask b and c from social media. the approaches are evaluated on offenseval@semeval2020 dataset. the given tweets are pre-processed and vectorised using tf-idf in machine learning models. the classifiers namely multinomial naive bayes, support vector machine, logistic regression and random forest were employed to build the models for sub tasks b and c. we have employed bert model to build the model for all the subtasks in the three languages. logistic regression performs better results for the subtask c in english. bert model give better results for subtask a in the english, danish, greek and subtask b in english respectively. our models outperform the base line for all the three tasks. the performance may be improved further by evaluating different models with the adoptable features.