IIITG - ADBU at S em E val-2020 Task 12: Comparison of BERT and B i LSTM in Detecting Offensive Language (2020)

Abstract: Task 12 of SemEval 2020 consisted of 3 subtasks, namely offensive language identification (Subtask A), categorization of offense type (Subtask B), and offense target identification (Subtask C). This paper presents the results our classifiers obtained for the English language in the 3 subtasks. The classifiers used by us were BERT and BiLSTM. On the test set, our BERT classifier obtained macro F1 score of 0.90707 for subtask A, and 0.65279 for subtask B. The BiLSTM classifier obtained macro F1 score of 0.57565 for subtask C. The paper also performs an analysis of the errors made by our classifiers. We conjecture that the presence of few misleading instances in the dataset is affecting the performance of the classifiers. Our analysis also discusses the need of temporal context and world knowledge to determine the offensiveness of few comments.

## introduction detecting offensive language in social media is gaining a lot of importance. it is becoming commonplace to encounter offensive content in social media. this type of content if left unattended has the potential of creating a lot of damage to society. as such, research has been directed at developing automated systems for the detection and removal of offensive posts. offenseval 2020 is a shared task organized as part of the international workshop on semantic evaluation 2020 (semeval 2020) @xcite to develop systems for detecting offensive content. offenseval was also organized as part of semeval 2019 @xcite . offenseval 2020 had 3 subtasks: (1) subtask a required participants to determine if a given tweet is offensive or not, (2) for offensive tweets, subtask b required determining whether the offense is targeted or untargeted, and (3) if the tweet is a targeted offensive tweet, subtask c required determining if it targets an individual, group, or other. this task was held for arabic, danish, english, greek, and turkish language. we participated in all three subtasks for the english language. in this study, we used the bidirectional encoder representations from transformers (bert) @xcite and bidirectional long short-term memory (bilstm) @xcite classifiers. ## related work the history of automatic offensive language detection goes back to @xcite , where a rule based method was used to detect flames in comments received through the feedback forms of web pages. this topic has gained much research attention in recent times. apart from offensive language, research has been performed to detect hate speech, abusive language, cyberbully, profanity, insults, and aggressiveness. @xcite defines these related terms. waseem and hovy (2016) used a logistic regression classifier to detect racist and sexist tweets. logistic regression and multi-layer perceptron classifiers were used in the study performed by @xcite . this study was performed to detect insults in wikipedia comments. @xcite differentiated between hate speech and offensive language. it was found that differentiating between hate speech and offensive language is difficult. almost 30% of the hate speech was misclassified as offensive language. @xcite used an svm classifier to perform this same 3-class classification and obtained an accuracy of 78%. using ensembles and meta-classifiers, @xcite was able to improve the accuracy to 79.8% on this same task. ## data the semi-supervised offensive language identification dataset (solid) dataset was used in offenseval 2020 for the english language task. this dataset is discussed in detail in @xcite . ## results on closer analysis of other comments containing the above mentioned words (sick, boring, etc.), we found that many comments were labelled as offensive in the data set although the comments seemed to be not offensive. such type of comments is listed in we, thus, conjecture that the presence of such misleading instances in data as the cause of wrong predictions for the instances mentioned in ## conclusion in our study, we found that both bert and bilstm performed equally well in subtask a and c during the validation stage. in subtask b, bert had a gain of 3% in the f1 score compared to bilstm. in the official results, our classifiers obtained an f1 score of 0.907, 0.653, and 0.576 in subtask a, b, and c respectively. error analysis showed the presence of a few misleading instances in the dataset. we conjecture these instances to be affecting the performance of the classifiers. the classifiers also failed to classify tweets that required temporal context and world knowledge to interpret them. incorporating world knowledge to help detect offensive content is a possible direction for future work.