NUIG at S em E val-2020 Task 12: Pseudo Labelling for Offensive Content Classification (2020)

Abstract: This work addresses the classification problem defined by sub-task A (English only) of the OffensEval 2020 challenge. We used a semi-supervised approach to classify given tweets into an offensive (OFF) or not-offensive (NOT) class. As the OffensEval 2020 dataset is loosely labelled with confidence scores given by unsupervised models, we used last year’s offensive language identification dataset (OLID) to label the OffensEval 2020 dataset. Our approach uses a pseudo-labelling method to annotate the current dataset. We trained four text classifiers on the OLID dataset and the classifier with the highest macro-averaged F1-score has been used to pseudo label the OffensEval 2020 dataset. The same model which performed best amongst four text classifiers on OLID dataset has been trained on the combined dataset of OLID and pseudo labelled OffensEval 2020. We evaluated the classifiers with precision, recall and macro-averaged F1-score as the primary evaluation metric on the OLID and OffensEval 2020 datasets. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/ .

## introduction we are generating data at a rate which has never been seen before 1 , while social media plays a big role in our everyday life. most people with smartphones have a social media account and an active social media life. however, due to anonymity and freedom of expression, social media has become a breeding ground for offensive content. this content can be deemed as offensive if it intends to demean a person or a group of people due to their ethnicity, sexual orientation or by a personal attack 2 . this raises the need for an evaluation system that could automatically detect such offensive text. sub-task a (english only) in the offenseval 2020 @xcite provided the offenseval 2020 dataset @xcite of english tweets, which has been labelled with average confidence and standard deviation of the confidence scores, e.g. "@user his ass need to stay up" has an average confidence of 0.833496 while standard deviation is 0.140625. @xcite used an unsupervised learning approach to produce these confidence scores. since the system is evaluated on its label prediction performance (offensive off, not-offensive not) and not on these scores, it was up to the participant to decide how to best map the scores to labels. we trained different text classifiers on the olid @xcite dataset to predict the labels of the offenseval 2020. this pseudo-labelling approach avoids the efforts involved in the manual adjustment of threshold values of average confidence (avg conf ) and average standard deviation (avg std). ## related work there has been significant research on aggression detection, hate speech detection @xcite @xcite and cyberbullying @xcite . based on this this work is licensed under a creative commons attribution 4.0 international licence. licence details: http:// creativecommons.org/licenses/by/4.0/ . 1 https://www.forbes.com/sites/bernardmarr/2018/05/21/how-much-data-do-we-create-every-day-the-mind-blowing-statseveryone-should-read/#75d1769a60ba 2 https://dictionary.cambridge.org/us/dictionary/english/hate-speech parameters stats the number of tweets/samples 9,075,418 average word count per tweet 15.64 average functional words count per tweet 5.61 average hashtags per tweet 0.08 hybrid approaches which combine a recurrent neural network (rnn) and a cnn have been proven to be better at text classification. @xcite has designed such an approach for sentiment analysis. in their research, they combined multiple outputs of a convolutional filter to form a vector representation of text, which was then fed to a bilstm. dynamic meta-embedding (dme) and contextualised dme (cdme) introduced by @xcite has shown significant improvement in a variety of natural language processing tasks such as natural language inference, sentiment classification and image-caption retrieval. pre-trained bidirectional encoder representations from transformer (bert) @xcite based models (bert) have performed exceptionally well in many natural language processing tasks such as text classification, natural language generation and machine translation. we designed our experiments based on cnn + bilstm, cdme, dme and bert. ## methodology in this section, we are giving insights about the methodology followed to process and pseudo-label the data in subsection 3.1 and subsection 3.2 respectively. ## experiments we outline the experiments performed on the olid dataset and the combined dataset offenseval 2020 + olid in subsection 4.1 and subsection 4.2 respectively. ## results we have divided this section into two subsections. in the subsection 5.1, we report on results achieved on the olid dataset. subsection 5.2 discusses the performance of bert on the offenseval dataset. ## conclusion with an macro-averaged f1-score of 0.89 using bert on the offenseval 2020 test set, we showed that pseudo-labelling can help to create a model trained on a loosely labelled dataset. the process of training the model on the small dataset, before training on a similar, yet bigger dataset, is more time efficient compared to a manual annotation. manual adjustments of the thresholds is further not straightforward and it is rather difficult to analyze larger datasets manually. pseudo-labelling solves this issue by automatically annotating the data. nevertheless, pseudo-labelling is just one approach with which this issue of unlabelled data could be solved. k-nearest neighbour (knn) with two clusters can be used to label the data as well. for our future work, we will concentrate on feature engineering, focusing mainly on the uniform resource locators (urls). as urls in the tweets were removed while processing the data, we think that they could be useful if they are normalised and considered as a token.